

STATE OF NEW MEXICO 
PUBLIC EDUCATION DEPARTMENT 

300 DON CASPAR 
SANTA EE, NEW MEXICO 87501-2786 
Telephone (505) 827-5800 
www.ped.statc.nm.us 



DR. VERONICA C. GARClA 



BILL RICHARDSON 
Governor 



SECRETARY OF EDUCATION 



August 31, 2009 
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1000 Thomas Jefferson Street NW 
Washington, DC 20007-3835 

Dear Dr. Cohen: 

The PED received student test data files for the New Mexico Alternate Performance Assessment that contained two 
sets of data errors produced by AIR in the most recent reporting cycle. In the first instance, on or about July 10, 
2009, the PED received a data file that contained miscodes for test completion status. This substantial error resulted 
in miscalculations in student proficiency rates. That is, students were included in calculations when they should have 
been excluded, and they were excluded from calculations when they should have been included. PED staff was 
informed of the errors by Dr. Werner Wothke, and PED staff applied corrections to the data and recalculated 
proficiency rales. The impact of the data corrections was substantial, with the percentage of students scoring 
proficient and above in the Students with Disabilities subgroup increasing by as much as seven percentage points at 
the district level. 

In the second instance, on Friday, August 21, 2009, the PED received notification from Dr. Werner Wothke that the 
student test data file that was delivered to the PED by AIR on July 10, 2009, contained additional coding errors for 
test completion status for 77 students. These data affected AYP calculations in eight schools across the state. AYP 
calculations at the subgroup and overall will be changed for the following 10 schools: 

De Vargas Middle School (Santa Fe Public Schools) 

La Academia de Esperanza (Albuquerque Public Schools) 

Cibola High School (Albuquerque Public Schools) 

Rio Rancho Elementary School (Rio Rancho Public Schools) 

Hayes Middle School (Albuquerque Public Schools) 

Shiprock High School (Central Consolidated Schools) 

Puesta del Sol Elementary School (Rio Rancho Public Schools) 

This second set of collections will require that PED staff and contractors again reanalyze the data and produce 
corrected reports. 

The student test data from the New Mexico Alternate Performance Assessment is an important source of student test 
score information that was used by the PED to determine the adequate yearly progress of district schools, charter 
schools, State education institutions, and U.S. Bureau of Indian Education Schools. The data was also used to 
calculate Achievement Gap statistics, an important report on the performance of New Mexico's race and etlinicity 
subgroups and program subgroups. The results of these analyses were communicated in a press conference by State 



Secretary of Education Garcia, they were published by media statewide, released in hardcopy reports, posted on the 
PED website, and had been prepared for submission to the federal government in a major data collection. Rework of 
this data has compromised a number of production deadlines and burdened our staff that is already challenged by a 
demanding workload. Further, schools sent letters to parents regarding AYP determinations and their right to choose 
their child's school based on those determinations. 

Though we appreciate full disclosure of errors in any deliverable associated with the New Mexico Alternate 
Performance Assessment, the timing of error discovery and disclosure and the exposure to multiple disclosures has 
been particularly disruptive and costly. By the time full recovery from these errors is achieved, PED professional 
staff will have dedicated an estimated 140 hours in evaluating the impact of the errors, preparing corrected data sets 
for analysis, reanalyzing data for the Achievement Gap and Adequate Yearly Progress, and preparing new reports, 
tables, graphs, and PowerPoint slides. Additionally, it is estimated that contractor costs for preparing corrected data 
for analysis and reanalyzing data will exceed $20,000. Additionally, the PED must replace its supply of 200-page 
hard copy color reports on AYP and the Achievement Gap for distribution to stakeholders. 

Please recall that the contract requires delivery of data without Contractor production errors and you provided 
assurances prior to the delivery of this year's data that a number of quality checks, including test completion codes, 
would be performed to avoid errors that have occurred in data files in prior years. Section 4b, Attachment 3-A of the 
Scope of Work in Contract Amendment Number 3 specifies that "failure to meet delivery deadlines and data 
accuracy requirements for the student pass/fail test data file shall result in a penalty of $2,500 per day for the first 
five days, and increasing to $5,000 per day thereafter (including weekends)." The computation of the penalty from 
July 10, 2009 until the date that the corrected data shall be submitted to the PED already exceeds the maximum 
penalty allowed under the contract amendment. Contract Amendment Number 3, Attachment 3-A specifies that "any 
penalties assessed against the Contractor pursuant to the terms of this amendment shall not exceed $25,000 in total." 
Therefore, the Agency is assessing a penalty of $25,000 for said Contractor production errors. Given the expenditure 
of agency and contractor resources, this amount now seems seriously inadequate. 

PED expects that the data will be corrected and submitted to the PED no later than September 8, 2009, so the PED 
and its contractors can perform, a second round of calculations and AYP determinations so Secretary Garcia can 
certify final AYP results on September 11, 2009. The school districts and the public will be astonished to learn that 
their AYP results were affected by data file errors. Secretary Garcia expects AIR to issue a written apology in the 
form of letters to each Superintendents affected by the data errors. 

Please submit a check in the amount of $25,000 to the Assessment and Accountability Division of the PED for this 
penalty. "We expect that ATR will make every effort to avoid future compromises of data quality. 

Sincerely s~\ 

Jf If) / 

Tom Dauphinee, PhD, Interim Supervisor 
Assessment and Accountability Division 



cc; Dr. Veronica C. Garcia, Secretary of Education 

Dr. Catherine Cross Maple, Deputy Secretary, Learning and Accountability 

Dr. Sheila Hyde, Assistant Secretary Quality Assurance and Systems Integration 

Beverly Friedman, Public Information Officer 

Dr. Cindy Gregory, Chief Statistician, Academic Growth and Analysis 

Robert Piro, Chief Information Officer 

Select PED Staff 
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April 7,2010 

Martin Borg 
President 

Measured Progress 
100 Education Way 
Dover, NH 03820 

Dear Mr. Borg: 

New Mexico school districts received test materials for the spring 2010 administration of the Standards Based 
Assessment (SBA) that contained a number of significant errors. These errors disrupted the administration of the 
SBA and caused considerable dismay and frustration among students and test administrators statewide. The PED is 
imposing a penalty for these errors, as the contract provides. 

Directions for Administration in English Grade 5 SBA in Writing, English Grade 5 Mathematics, English Grade 6 
Science, English Grade 7 Mathematics, English Grade 1 1 Science (two sessions), and Spanish Grade 4 Writing all 
contained errors directing students to the wrong pages to continue the test. Directions for Administration errors were 
first reported by Albuquerque Public Schools on March 23, 2010. Staff members from the Assessment and 
Evaluation Bureau of the PED and from the New Mexico Project staff at Measured Progress gathered information 
on errors and published a correction notice to districts on March 25, 2010, early in the SBA testing window. In order 
to correct the effects of these errors, test adnunistrators read scripts directing students to the correct pages of the 
SBA. Altogether, an estimated 4,660 test administrators adjusted procedures to communicate these changes to an 
estimated 1 16,500 students statewide. 

The Grade 4 Spanish SBA in Math contained an item without a correct response option (Form 2, Item 22). This 
error, reportedly attributed to desktop publishing, affected an estimated 850 test booklets. The error was reported by 
Bernalillo Public Schools on March 30, 2010. On April 1, 2010, the PED and Measured Progress notified school 
districts statewide of the error and assured the districts that students who attempted the item would be scored as if 
they answered that item correctly. 

Grade 5 English SBA answer documents for Form 2 contained out of order pages in 16 documents distributed in 6 
districts, Albuquerque Public Schools, Clovis Municipal Schools, Dulce Independent Schools, Los Alamos Public 
Schools, West Las Vegas Public Schools, and Rio Rancho Public Schools. These out of order pages affected die 
administration of two sessions in Reading and two sessions in Science. Rio Rancho Public Schools fust reported the 
error on March 24, 2010. The PED reported the error to Measured Progress and an investigation was conducted to 
determine the statewide scope of the error. Measured Progress staff used serial numbers and distribution records to 
identify schools that potentially received faulty answer documents. Measured Progress contacted the identified 
school districts to alert them to the error on March 30, 2010. 

The Grade 8 English SBA in Math (p. 63) contained a multi-part constructed response question with an extra letter 
"D" in the left margin covered with white tape. This tape was deliberately applied during the production process 



after the tests were printed and bound. Measured Progress explained that the extra letter that should not have 
appeared and was therefore covered. Unfortunately, in order to ship the test booklets to districts on time to allow 
inventory and redistribution of materials to schools, it was not possible to completely replace all the eighth grade 
English test booklets with new booklets with the correction. 

One test booklet for the Grade 8 English SBA in Science was missing pages 115 through 122. Dulce Independent 
Schools reported the error, and after consultation with Measured Progress on the same day, replaced the booldet 
with another on hand. 

Though we appreciated full cooperation of Measured Progress in investigating errors and preparing collection 
notices for school districts, the errors were decidedly disruptive to the administration of the SBA. The PED received 
complaints from school districts regarding the scope and number of errors. Districts and schools dedicated precious 
staff time to identify and report errors, check test materials, and execute corrective measures. Test Administrators 
and Coordinators are heavily overburdened during the SBA administration window, and disruptions due to these 
errors caused great frustration. The PED seldom receives complaints from students when errors occur. However, we 
did during this administration. Students find such errors distracting, confusing, anxiety provoking, and frustrating, 
adding to the stress of testing and potentially compromising their performance. Experiences such as tin's often lead 
students to question the purpose and value of testing in general. The PED is placed in the difficult position of 
defending testing to guide instruction and support school accountability reporting, purposes that are somewhat 
abstract for some students. 

Please recall that the contract requites delivery of test materials without Contractor production errors. Section lb, vii 
in Attachment 1-A of the Scope of Work in the Contract specifies that "failure to meet that standard in production 
and delivery of all assessment materials to any school district, charter school, State educational institutions, and U.S. 
Bureau of Indian Education schools shall result in a penalty of $2,500 per day for the first five days, and increasing 
to $5,000 per day thereafter (including weekends) until the correct materials are received." The computation of a 
penalty from the date errors were reported until the date that a remedy was provided in the instance of out of order 
answer documents, the longest period from report to remedy. The penalty for that production error totals $22,500. 
Therefore, tire PED is assessing a penalty of $22,500. 

PED expects that appropriate scoring procedures will be implemented to correct the aforementioned item errors such 
that students will not be penalized for the said errors. 

Please submit a check in the amount of $22,500 to the Assessment and Accountability Division of the PED for this 
penalty. The PED is planning a debriefing meeting with districts after the end of the 2010 SBA test window to 
provide districts and the PED opportunity to express concerns regarding all aspects of ordering, production, 
delivery, return shipment, and reporting of the SBA and New Mexico High School Competency Exam (NMHSCE). 
We expect that Measured Progress and subcontractor staff representing all organization divisions supporting those 
services will be present at that meeting in New Mexico. We further expect that Measured Progress and its 
subcontractors will make every effort to avoid future errors in the production and distribution of the SBA and 
NMHSCE. 




Interim Supervisor 

Assessment and Accountability Division 

cc: Dr. Veronica C. Garcia, Secretary of Education 

Dr. Catherine Cross Maple, Deputy Secretary of Learning and Accountability 
Beverly Friedman, Public Information Officer 
Select PED Staff 
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September 13, 2011 

Mr. Martin Borg 
President 

Measured Progress 
100 Education Way 
Dover, NH 03820 

Dear Mr. Borg, 

New Mexico school districts recently received hardcopy Student Reports for Parents from the 201 1 
Standards Based Assessment (SBA), and some student results in those reports contained errors. Those 
errors caused confusion and concern among teachers and administrators, and they delayed delivery of test 
results to parents more than 30 days. The PED is imposing a penalty for these errors under the terms of 
the contract. 

An estimated 10% of the hardcopy SBA Student Reports to Parents shipped to districts contained errors 
in the section titled, "A Closer Look at Your Child's Performance." In those reports, the performance 
indicators for Reporting Categories were printed with the wrong symbols. 

On August 19, 201 1 , the PED received a report of discrepancies between hardcopy and online versions of 
the Student Report to Parents from the Alamogordo Public Schools. Discrepancies were in grade 6 
reports in math. The PED immediately reported the finding to your company. Alamogordo contacted your 
company to report the errors on August 22, 201 1 . In response to Measured Progress' request, the 
Alamogordo schools returned all hardcopy Student Reports to Parents for examination. Your staff began 
investigating online reports, data files, and printing processes as potential sources of errors. On August 
23, 201 1, the PED received a preliminary report on the scope of the problem. Your staff verified that all 
PDF versions of the reports and the data files were accurate, and it appeared that errors were only in 
shipped hardcopy reports. On August 24, 201 1 , the PED directed the New Mexico Program staff to alert 
14 school districts with scores from the same source file as Alamogordo about potential reporting errors. 
Your staff recommended that those districts not send Parent Reports until additional information about 
errors was available. The PED contacted four school districts, requesting their assistance in identifying 
errors. On August 25, 201 1, Belen Consolidated Schools identified a small number of errors in math in 
grades 3, 5, and 6 from a sample of reports. Your staff also identified errors in math in those grades from 
internal sources. Under direction from the PED, the New Mexico Project staff sent an email message to 
district and administrators statewide explaining the errors and the accuracy of other scores, 



recommending that reports not be sent to parents, committing to replace defective reports, providing 
assurances for a rapid resolution, and providing a contact number to answer questions. On August 29, 
201 1, your staffs investigation into the scope and source of errors was still underway. Your staff did not 
have an estimated time for determining the scope of the problem, though they offered to print corrected 
Student Reports to Parents for all tested students and ship them to districts by September 2, 201 1 . The 
PED directed your staff to print and ship the corrected reports. 

Though we appreciated full cooperation from the New Mexico Project staff in investigating errors, 
sending notices to school districts, and printing and shipping corrected reports, the errors disrupted timely 
delivery of important student performance information to parents. Districts dedicated a substantial share 
of staff time to identifying and collecting defective reports, return shipping those to Measured Progress, 
and distributing corrected reports. They were required to do so while being exceedingly busy with a new 
school year. School districts also raised concerns about the integrity of the SBA Online Data Reporting 
System that was introduced in 201 1 . Though this system was not at fault for these errors, the incident 
reignited concerns that delayed its adoption a year and a half ago. It was also disconcerting that the source 
of the errors remained unidentified at the time of this letter. Lacking information about the source of the 
problem prevented replacement of defective reports only. As a result, school districts were required to 
twice handle reports for an estimated 1 71 , 244 students, instead of redistributing reports for one-tenth of 
that number. 

Please recall that the contract requires delivery of test score reports no later than August 1 without 
Contractor production errors. Your staff stated that the last corrected Student Reports to Parents were 
delivered to districts on September 6, 201 1 . Section lb, xiv in Attachment 1-A of the Scope of Work in 
the Contract specifies that "failure to meet delivery deadlines and the accuracy standard for the district, 
school and student score reports shall result in a penalty of $2,500 per day for the first five days, and 
increasing to $5,000 per day thereafter (including weekends and holidays) until correct reports are 
received." The computation of a penalty from the date accurate reports were due and the date corrected 
reports were received by all districts is $172,500. However, section lg, v in Attachment 1-A of the Scope 
of Work specifies, "Penalties assessed against the Contractor pursuant to the terms of the Agreement shall 
not exceed $150,000 in any fiscal year while the agreement is in effect." Therefore, the PED is assessing 
a penalty of $150,000. 

Please submit a payment in the amount of $150,000 to the Assessment and Accountability Division of the 
PED for this penalty. I expect a report on the error and results of the investigation, with a corrective 
action plan for preventing similar production errors in the future. I further expect that Measured Progress 
and its subcontractors will make every effort to avoid future errors in the production and distribution of 
the SBA. 



Best Regan 



Hanna Bkafraera 

Secretary'of Education-designate 




HS/TD 



Cc: Cathie Carothers, Deputy Secretary of Policy and Indian Education 
Dr. Pete Goldschmidt, Director of Assessment and Accountability 
Dr. Tom Dauphinee, Deputy Director of Assessment and Accountability 
Willie Brown, General Counsel 



